Declarative Attention inside vLLM — 4 interactive diagrams

Strategy patterns → what blocks / the block table / metadata actually are → the "neutral middleman" → why the DA hook is surgical. All diagrams animated; use the buttons.

1 · The model is a skeleton; vLLM injects the plugins

Layers stack bottom→top. Every grey slot is an interface: the model's config.json fixes the shape (heads, window, hidden size); vLLM picks the implementation at load time. Try the buttons on the right.
▲ logits · LM head (ParallelLMHead)
▲ token ids + positions → VocabParallelEmbedding
model config
All layers are global attention → one backend everywhere.
get_attn_backend() → AttentionImpl
Strategy pattern: same Attention interface, kernel swapped by head-size / hardware / window.
quant_config → LinearMethodBase
The MLP math (gate_up → act → down) is fixed by the model; the GEMM kernel is injected.
CustomOp dispatch (norm + residual)
Residual adds are fused into RMSNorm by a torch.compile pass — model code never mentions the residual stream.

2 · Blocks, the block table, and "which KV exists"

blockFixed-size chunk of KV cache: K+V vectors for 16–32 tokens (4 here for display). The kernel can only read whole blocks.
block tablePer-request array: logical block # → physical block id. The kernel's only map of the request's memory. Blocks not listed are never read.
physical KV cacheGPU HBM carved into fixed slots. A request's blocks live scattered wherever the allocator put them.
commit (H2D)The row is staged in CPU numpy, then copied Host→Device. Edits are cheap while staged.
CPU · host — easy to edit

logical tokens

block table row (numpy staging)

commit() — H2D copy ▼
GPU · HBM — kernel reads here

physical KV cache (64 slots)

When a logical block fills, the allocator grabs a free physical slot anywhere and appends its id to the row (append_row). Hover a table entry → the matching physical block pulses. The kernel walks only these ids — this list is "which KV exists for this request".

3 · The metadata path — the neutral middleman

Every decode step, before the kernel runs, vLLM assembles a metadata bundle and hands it over via a scoped context. The model's forward signature never sees it. The dashed orange region is the only place DA touches.
SchedulerOutputwhich reqs run,
how many tokens
→
vLLM metadata path (middleman) — DA hook lives here
BlockTableCPU numpy staging
append_row()
→
commit()H2D copy to
block_table_tensor
→
MetadataBuilderFlashAttentionMetadata
{block_table, seq_lens,
query_start_loc, slot_mapping}
→
set_forward_contextscoped global,
not an argument
→
Attention layerget_attention_context
(layer_name)
→
FlashAttentionImplflash_attn_varlen_func
(…, block_table=…)
The packet cycles left→right. Model code (left of the dashed box, and the layer itself) and the kernel (right) are untouched by DA — the hook rewrites the staged row between append_row() and commit().

4 · The DA hook, animated: tags trapped in the middleman

Press play. The model streams tokens; the state machine traps tags at the closing >; the staged block-table row is rewritten before commit; the kernel simply reads fewer blocks. Prompt = 12 magic chunks (2 blocks each) + scaffold; response grows as it generates.

model response stream (its own words = control plane)

DA state machine — tag buffer

GLOBAL
FOCUS
LOCAL

three bands — where the edit lands

MODEL CODE
registry class + weights · untouched
METADATA PATH DA HOOK
block-table row rewritten per step, pre-commit
KERNEL
flash_attn_varlen_func · untouched

blocks read per step

—

the request's block-table row (what the kernel will be allowed to see)

scaffold + sink (always kept) magic chunks C1–C12 response so far (always kept) masked — not in the row the kernel gets
speed
The surgical part: the model's math and the kernel's algorithm never change. One CPU-side row edit, at the one place where "which KV exists" is decided — that's the entire hook. Watch FOCUS drop the row from 28+ blocks to ~8, and LOCAL to ~6, while the response (purple) and scaffold (blue) stay put.